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EXECUTIVE SUMMARY 



The 1993-94 Integrated Learning System (I.L.S.) project 
involved approximately 70 schools in almost every New York City 
community school district. I.L.S. is a means of delivering 
individualized instruction in various curriculum areas through a 
network of computers. 

To help schools learn about and operate the computers and 
software in an I.L.S., districts were given the option of hiring 
companies that were experts in the use of such systems. 
Previously , the Board of Education of the City of New York had 
reviewed proposals and selected four companies (referred to as 
educational systems integrators) from which schools and districts 
might choose; Instructional Systems Inc., Jostens, the Waterford 
Institute , and Titan. Of the four integrators , Titan elected to 
have the Office of Educational Research (OER) evaluate ita 
program. 

Titan , which is based in Colorado, was chosen as integrator 
by six schools. Titan, in turn, contracted with Computer 
Networking Specialists (C.N.S.) on Long Island to perform the 
integration services for them, and with the Waterford Institute 
to provide teacher training. Of the six schools which chose 
Titan as integrator, two were part of the grantback phase and the 
other four were in the capital phase of the project* The 
grantback phase began close to schedule, with computers delivered 
in early autumn 1993 or before the school year started. Schools 
?n the capital phase, however, did not begin using their 
computers until 1994. A variety of problems resulting from the 
asbestos crisis in New York City public schools, as well as 
delayed deliveries and installations affected both phases of the 
project, but especially the capital phase. 

Schools were unanimous in their opinion that the C.N.S. 
representative was professional, knowledgeable, responsive to 
problems and needs, and a pleasure to work with. However, the 
length of time it occasionally took C.N.S. to get to one of the 
schools to fix a system problem was a concern. 

Half of the schools were very satisfied with the teacher 
training they received from the Waterford Institute, while the 
other half voiced dissatisfaction with the initial training in 
several areas: the amount of time allocated for training; the 
topics covered; and, in one case, the trainer's manner of 
presenting the topics. 

Trainers from C.N.S. or Waterford spent a good deal of time 
with the schools 1 lab managers to ensure that they could operate 
the system software, enroll students, change placements, etc. As 
the project was implemented, teachers relied on these lab 
managers, since they were the ones trained to use the system. 



Opinions about the software programs were mixed. Although 
many teachers thought the lesson software was beneficial to 
students, others said students found it boring, A considerable 
number of teachers said that the software did not fit their 
schools 1 curricula, so that they had difficulty integrating it 
into classroom activities. 

one area of dissatisfaction was the schools 1 involvement in 
decision making about the I.L.S. project. Most school 
administrators reported that they had little or no input into 
initial decisions about what integrator or software to use, and 
teachers had even less. 

Student achievement scores showed no significant differences 
in reading between program participants and the rest-of-school 
population. Mathematics scores indicated a trend towards 
differences in grade 3. However, this trend favored the rest-of- 
school population, not program participants. 

Based on these findings, OER makes the following 
recommendations : 

Reexamine the teacher training process to clarify such 
aspects as how much and what kinds of training teachers 
need before students begin to use the I.L.S. system, 
and what kind of ongoing support teachers need to 
ensure successful student use of the lab. 

Clarify more completely the roles of C.N.S. and 
Waterford in delivering technical and educational 
support to schools. 

Consider how the program expects schools to integrate 
use of the I.L.S. lab with the rest of their curricula, 
and how systems integrators will work with schools to 
effect this integration. 
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I. INTRODUCTION 



INTEGRATED LEARNING SYSTEM PROJECT 

An Integrated Learning System (I.L.S.) is a means of 
delivering individualized instruction in various curriculum areas 
through a network of computers. Individual microcomputers are 
connected to a central microcomputer (also called a file server) , 
controlled by a teacher or computer lab manager, which sends the 
programs students use to their separate computers. A computer 
network can simply provide a variety of software for users to 
access. In an I.L.S. , however, the system software plays an 
integral role in managing students 1 progress through skill 
levels. The software typically tests students before they begin 
the software lessons to provide initial placement at the correct 
skill level. It then delivers lessons to students, assesses 
their performance, and adjusts their levels accordingly. 
Lab managers, or teachers, can at any point change students' 
levels, select skill areas or subject areas for software lessons, 
and change the sequence in which the lessons are presented. 
I.L.S. which are intended to benefit students of elementary 
school age, provide a number of reports, including profiles of 
skills covered by a class and progress for individual students, 
which teachers can use to help them make decisions about student 
placement and what they want the software to cover. 

The 1993-94 I.L.S. project involved approximately 70 schools 
in almost every New York City community school district. Almost 
every school district elected to place their computer labs in 



3 



elementary schools, but a few put them in intermediate schools to 
be used by fifth or sixth graders* 

The project proceeded in two phases. The first, financed by 
funds from the federal Chapter 1 program/ was called the 
"grantback" phase and was scheduled to begin at the beginning of 
the 1993-94 school year. The second, with funds from the New 
York City Council, was known as the "capital" phase and was 
intended to begin later in the fall of 1993. 

To help schools learn about and operate the computers and 
software involved in an I.L.S., districts were given the option 
of hiring companies that were experts in the use of such systems 
(hereafter referred to as educational system integrators) who 
performed a variety of services. Their roles included 
coordinating the delivery and installation of computer equipment 
and security devices, installing the I.L.S. software, training 
teachers and other school personnel in using the computers and 
software, troubleshooting problems, and continuing to help school 
personnel throughout the year. The Board of Education of the 
City of New York, in the previous school year, had reviewed 
proposals and selected four integrators from which schools and 



'Chapter 1 is a federal funding source for remediation programs 
that address student needs in basic reading , writing, 
mathematics, and English-language skills. A school is eligible 
for Chapter 1 funds if its percentage of low income students is 
equal to or greater than the citywide average based on a formula 
which calculates students' eligibility for free lunch and Aid to 
Families with Dependent Children (A.F.D.C.). Students are 
eligible for Chapter 1 remediation programs when they score below 
the state reference point on standardized tests. 

2 
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districts might choose: Instructional Systems Inc., Jostens, the 
Waterford Institute, and Titan. 

As part of its contract, each integrator was required to 
provide an evaluation of their services during the 1993-94 school 
year. Integrators were given the option of hiring their own 
evaluators or of having the Office of Educational Research (OER) 
of the Board of Education of the City of New York conduct the 
evaluation. 

Of the four integrators, Titan elected to have OER evaluate 
it. Titan, which is based in Colorado, was chosen as integrator 
by six schools. Titan contracted with Computer Networking 
Specialists (C.N.S.) of Long Island to perform the integration 
Services for them, and with the Waterford Institute to provide 
teacher training in the six schools. This report, therefore 
discusses the performance of C.N.S. ami the Waterford Institute 
in their fulfillment of the various integrator roles. 

To clarify the ensuing discussion, some description of 
I.L.S. software is necessary. Some I.L.S.s use software 
specially developed for them; thc.t is, all of the software 
lessons, and the software that manages students 1 progress through 
the program, was developed by one source. Jostens is one example 
of this kind of system; Instructional Systems, Inc. which is a 
franchise of the Computer Curriculum Corp. and uses only its 
software, is another. 

C.N.S. uses what is known as a more "open" system, because 
it uses parts of commercially available software and then creates 



the management system that links the lessons together. C.N.S. 
thus has the ability to use a variety of software in its system. 
The Waterford Institute helps schools use whatever software the 
schools decide upon. In their role as subcontractor to Titan in 
this project , Waterford was responsible for training teachers in 
schools which had chosen to use the C.N.S. system. 
EVALUATION METHODOLOGY 

Of the six schools which chose Titan as an integrator, two 
were part of the grantback phase and the other four were in the 
capital phase of the project. The grantback phase began close to 
schedule, with computers delivered in early autumn 1993 or, in 
some cases, even before the school year started. Schools in the 
capital phase, however, did not begin using their computers until 
1994, in some cases not until March or April. A variety of 
problems resulting from the asbestos crisis in New York City 
schools delayed deliveries and installation, and affected both 
chases of the project, but especially the capital phase. 

Because the two groups of schools had access to the labs for 
different periods of time, this evaluation treats the two phases 
of the project differently. Topics covered for the two grantback 
schools encompass the entire school year and included the 
following: the installation and setup of equipment and software; 
initial training of school staff; how the labs have been used 
throughout the year; ongoing work with integrators; general 
satisfaction with all aspects of integrators' performance; school 
personnel's satisfaction with the I.L.S. software and their 
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perceptions of how it affected students; and academic achievement 
attributable to use of the system. Since the four capital 
schools had only a short time to begin using their labs, however, 
this evaluation covers only lab setup and installation, initial 
training and subsequent work with integrators, use of the labs to 
date, problems that have occurred, and school personnels 1 
perceptions of (or hopes for) benefits to be derived from using 
the system. No conclusions about students' academic achievement 
or overall satisfaction with integrators 1 work are drawn for the 
capital schools. 

Furthermore, the two schools in the grantback phase of the 
project were dissimilar in important ways. One was an 
alternative minischool for a small number of middle school-age 
students who had had problems in their original schools, while 
the other was a much larger elementary school in which 14 classes 
of upper elementary-age students used the lab, The^e two schools 
are therefore described first in Chapter II of this report. The 
four schools in the capital phase were all elementary schools 
using the labs for a range of ages, and are discussed together 
later in the chapter. 

In conducting this study, evaluators: 

• reviewed relevant program documents, including those from 
the Office of Technology of the central Board and from 
integrators, 0 

• observed students using I,L,S, labs; 

• interviewed administrators, teachers, and lab managers 
in project schools, as well as C.N*S. and Water ford 
personnel ; and . 

5 
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♦ administered questionnaires to teachers, lab managers, and 
administrators in project schools. 

SCOPE OF THIS REPORT 

Chapter I of this report describes I.L.S. and the 1993-94 

project in New York City public schools, as well as the 

methodology for evaluating those schools in which Titan was 

chosen as the educational systems integrator. Findings for the 

two schools in the grantback phase of the project, and for the 

four capital phase schools, are presented in Chapter II. Chapter 

III offers conclusions and recommendations. 
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II. FINDINGS 

SCHOOL ft 

Description of th e School 

School A is an alternative school for middle school students 
with emotional problems that prevent them from succeeding in 
traditional school settings. In May 1994 there were 57 students 
on record. According to the school's director, students average 
between three and six months at the school and are then 
mainstreamed back to their original schools, and almost all of 
them go on to graduate. The director also said that students 
showed a marked increase in reading levels while at the school. 

The school has six teachers, each of whom teaches more than 
one subject to different grade levels. Three of the teachers 
elected to use the new computer lab, and they worked in the 
computer lab with all of the students in the school. The school 
has a program in which eighth graders leave the school at 11:00 
a.m. on many days for occupational studies, such as working at a 
nearby senior center. Because of this, eighth graders used the 
computer lab much less than other students. 

A teacher new to the school acted as the de facto computer 
lab manager, although she was a regular classroom teacher who 
covered language arts and mathematics for seventh graders. She 
was able to do this because of the small size of the school and 
because she had an extensive previous background working with 
computers. In fact, one of her primary reasons for coming to the 
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school this year was the opportunity to work with the new 
computer lab. 

Installation and O peration of the I.L.S. Lab 

According to the director, the district office made all the 
decisions about purchasing the lab, including the choice of 
integrator and software. The school was at first pessimistic 
about someone from the. outside coming in and installing software 
without consulting with the school, but because of the personnel 
from C.N.S. and Water ford the project worked out excellently. 
The director reported no problems with hardware or software 
installation, which he called "a phenomenal effort by C.N.S.," 
which coordinated all the details beautifully. The director had 
nothing but the highest praise for the contact person from C.N.S. 
and the trainer from Water ford for their efforts throughout the 
year. 

The I.L.S. lab opened in October 1993. It contains 20 color 
Macintosh computers with CD-ROM drives and a few printers 
(including one laser printer). A few additional Macintosh 
computers in the room were purchased with other funds. Since 
October, the eighth grade has used it once or twice a vreek, 
because they are out of the school a substantial portion of the 
time. The seventh grade has used it every day, and twice on most 
days, for periods of about 40 minutes. The seventh grade has 
used mathematics, language, and writing software, while the 
eighth grade has used math and language programs. 
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Staff Development and Training 

Teachers who were to use the computer lab, plus the 
director, received three full days of initial training before 
students started coming to the lab. This training consisted of 
an introduction to computers and their terminology, the use of 
the computer operating system, an understanding of and experience 
using some of the C.N.S. lesson software, and using the I.L.S. 
management software for enrolling students and placing them. 

Teachers reported a high degree of satisfaction with all 
aspects of the initial training, including the amount of time 
spent, the material covered, the staff developer's manner of 
presenting the material, and the printed material they received 
with the training. Because the training began before the 
school's computers were fully installed, one day was spent at the 
Waterford Institute's Manhattan office, where teachers said they 
enjoyed being able to concentrate on learning away from the 
school. The teachers and the director had high praise for the 
Waterford staff developer, both in the initial training period 
and through the rest of the year. 

The teachers additionally noted that the training was 
successful in part because they had such a small group of 
teachers using the lab, and were able to benefit from the 
experience of the one highly experienced teacher mentioned above, 
who was able to help the other two teachers at all points. 
Without her, the director said, it might have been difficult to 
succeed. This teacher herself suspected that the initial 

9 
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training by itself might not have been enough for someone 
unfamiliar with computers. Another teacher agreed that she "got 
us over some of the rough periods" in beginning to use the 
system. Nevertheless, he too felt that the three full days of 
training provided a solid background. 

After the initial training period, according to the 
teachers, the C.N.S. contact was primarily responsible for 
working with them to continue learning about the software, the 
management system, and student placement in the system • The 
Waterford staff developer helped throughout the year with 
curriculum-related issues, such as breaking down the curriculum 
to make it fit better with classroom instruction, and with using 
software tools such as a word processor and spreadsheet. 
Teachers reported that they spent much time, during the initial 
training and after, learning computer applications that would 
help them professionally, such as using a database program to 
file information about their students, and using a word processor 
to write papers and report cards. 
Curriculum Conne ctions 

Teachers in school A reported that, with the help of the 
staff developer from Waterford, they had progressed in connecting 
what their students did in the computer lab with their regular 
classroom curriculum. One teacher said "we've gone through and 
made a syllabus based on what we teach in CIMS* math and 

*The Comprehensive Instructional Management System (CIMS) 
mathematics project is a teacher-developed mathematics program 
for kindergarten through grade seven that includes teachers 
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language arts* We now link software lessons we select with CIMS 
lessons." Another teacher reported that the staff developer had 
"shown us how to break down the curriculum and take out pieces of 
the software to match it, setting up modules for what I teach." 
Both thought that the software now fit well into their own 
curriculum. 

Personnel at School A did have some criticism of the I.L.S. 
software. One teacher said he would have liked to have science 
and social studies software. Another commented that the system 
contained software that seemed to be made for younger children 
than those they taught. Like the earlier comment by the director 
that the school was at first suspicious of outsiders coming in 
and making decisions about what kind of system and software the 
school would use, these comments reflect the lack of 
collaboration between the school and the integrator in making 
decisions about the schools desires and needs. The director 
felt that "it would have been better to revise the software 
curriculum beforehand to make it easier to integrate into our 
curriculum." Nevertheless, teachers and the director felt that 
the software proved to be successful in their school, largely 
because of the efforts of C.N.S. and Waterford personnel, and 
also because the skills covered in the software were ones that 
students needed. 



manuals, student workbooks, criterion-referred tests, and a 
computerized test scoring and reporting system for managing 
mathematics instruction* 



A key feature of an I.L.S. is its ability to assess 
students 1 abilities and keep track of their progress* After 
eight months of using the lab, teachers at School A had not yet 
begun to make use of the system's reporting function, and some 
had not yet heard of it. The teacher who knew the most about 
computers did know the function existed but had not deemed it 
necessary to use it yet, since the teachers and students, in her 
opinion, had enough to do learning how to use the software. 
Teachers did know about the system's assessment capabilities, but 
did not think much of them. As one teacher explained, "The 
system does a little pre- and posttesting, but it doesn't really 
do assessment. Teachers decide at what level to let kids move 
ahead. " 

Perceptions of Benefits 

Overall, teachers at School A perceived a number of benefits 
for their students in using the I.L.S. lab. One was students' 
enjoyment. Students liked using the software, and, according to 
teachers, liked the idea of using computers: "Not pulling out 
paper and pencil is wonderful to them." Students enjoyed editing 
their writing with a word processor rather than correcting it on 
paper: "They 1 re a lot more creative sitting in front of a 
computer. H One teacher reported that students did not regard 
their use of the lab as a subject-area class. Instead, to them 
it was a "computer class." They also had an easy time learning 
to use the software. After the staff developer spent some time 
introducing the hardware to them, most of them felt competent to 
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use any of it on their own* One teacher cautioned, however, that 
his students 1 enjoyment was related to their ability 
levels — these students tended to get frustrated easily if the 
software was not matched carefully to their competence. 

Teachers mentioned other benefits of using this kind of 
software for their students. It prepares them for the future by 
teaching them to use and understand the workings of a computer. 
Using the numeric pad is useful for future job skills. Teachers 
also thought that, in general, the interactive drill, especially 
the math drill, was academically beneficial. 

Teachers at School A also seemed pleased with their own use 
of the computers. They enjoyed learning to use them, learning 
useful applications, and using them with their students. One 
teacher commented proudly that he had previously been "just about 
computer illiterate, and now I can hold my own." Teachers 
reported using the computers to set up databases, do students 1 
report cards, and network with their district's administrative 
database. 

Academic Achievement 

School A is an alternative school that acts as a 
transitional program for students with emotional problems. 
Students enter the program at different times and stay between 
three and six months before returning to a regular school 
setting. Therefore, achievement tests were not available to 
assess student achievement. 

13 
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SCHOOL B 

Setup and overview of Lab Use 

School B is an elementary school with four or five classes 
in each grade from kindergarten to fifth. Fourteen classes , from 
third to fourth grade, used the I.L.S. lab. Teachers reported 
that they had no prior knowledge of, or involvement with, 
decisions concerning the lab. They were told by the school 
administration that they were to participate. 

All classes had between 25 and 30 students, and in all cases 
the classroom teacher stayed in the lab with the class. In 
addition, a lab manager, who was not a regular teacher, was in 
the lab at all times. Classes came to the lab either two or 
three times per week, for 45-minute sessions. 

According to the lab manager, there were no problems with 
the delivery or installation of computer equipment and software. 
The lab of 25 Macintosh color computers was fully set up, 
secured, and being used by December 1993. The lab manager 
reported that she installed no other software on the system 
besides the C.N.S. software. Of the C.N.S. software, classes at 
School B used primarily reading and math software modules, and a 
few also used social studies lessons. 
staff De velopment and Training 

The lab manager reported that she received three full days 
of initial training (before classes started using the lab) from a 
Waterford staff developer. This training including using the 
system software, setting up classes and enrolling students in the 
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system, troubleshooting hardware problems like system failures , 
and generating reports. She was highly satisfied with all 
aspects of this training, including topics covered, the length of 
the training, the staff developer's manner of presenting the 
material, and accompanying printed handouts. She felt the 
initial training covered all areas that she needed in order to 
get started with classes. 

Teachers were not as satisfied with their initial training. 
They all reported receiving two 2-hour sessions of training, but 
nearly all felt that this was not sufficient for them to begin 
working with their students in the lab. They reported that 
almost all of this time was spent using the educational ■ software 
on the system , but that there were other topics they wished had 
been covered, such as basic computer operations, writing with the 
computer, starting and operating the computer system, and 
troubleshooting problems that might come up. Most felt they were 
not prepared enough, or comfortable enough , to use the system and 
the software well. One teacher said that she did not understand 
the programs. Another thought that she had enough training to 
begin working with students, but only barely: "We could not 
operate the computers or understand the programs." Altogether, 
teachers gave low ratings to all aspects of the initial training: 
the time allocated for training, the topics covered, the staff 
developer 1 s manner of presenting the material, and the printed 
material they received. Almost every teacher also thought that 
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their students had not received enough initial training to 
prepare them to work with the software. 
Ongoing Assistance and Cu rriculum Connections 

All teachers reported that they continued to meet with the 
staff developer weekly after their students had begun using the 
lab mostly to learn more about the software that the students 
were using. They also all said that they attempted to connect 
the instruction students received in the lab with their classroom 
instruction, for instance, by making classroom social studies 
lessons tie in with social studies software lessons in the lab, 
The lab manager provided help in making these connections. All 
the teachers felt, however, that the software did not fit well 
into their classroom work. They felt there was not enough 
software in the content areas they covered in class and that its 
topics were not well coordinated with their curriculum. 

The lab manager reported that she did not need much help 
after the initial training from the staff developer, who 
subsequently spent most of her one day a week at the school 
working with the students in the lab. Sometimes the staff 
developer also worked with the school's computer teacher, who 
worked in the I.L.S. lab only one day a week. 
Operation of the I.L.S. System 

According to the lab manager, system problems would occur 
from time to time. Sometimes, for instance, the enroll module 
froze (stopped operating) in the middle of operation. A few 
times the entire system crashed (stopped functioning) and she was 
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unable to get it started again. The lab manager indicated that 
she was somewhat concerned about the integrator's responsiveness 
to such problems, since it took them from a couple f days up to 
(at least on one occasion) two weeks to get to the school and fix 
the problem. 

The lab manager said that she regularly used the report 
function of the software, generating what the system calls 
"lesson reports by student, " telling what lessons individual 
students had covered and how well they did on them and how long 
they spent on each. Teachers indicated that they received these 
reports, but with few exceptions they did not find them very 
useful, primarily because they felt the software did not 
coordinate well with their own curriculum. 

A majority of the teachers indicated that they were involved 
during the year with placing their students in the system, or in 
changing their placement. None had any problems with the way the 
I.L.S. software handled student placement and progress through 
the lessons. The lab manager reported that she discussed 
changing student placements with the teachers, and that she would 
then make the agreed upon changes in the system software. 
Benefits for Students and Ongoing Issues 

Several teachers thought that the way the I.L.S. software 
drilled and reviewed skills was good for students 1 individual 
progress. Nearly as many other teachers, however, thought the 
program's academic benefits were limited because the software was 
not coordinated with their curriculum. One teacher commented 
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that the software could be beneficial if it were "coordinated 
with curriculum, not just dumped on us. 11 

Several teachers and the lab manager indicated that the 
students found the software boring, that there was too much 
repetition, and that the software tended to stay on the same 
lesson too long. One teacher said her students initially looked 
forward to using the computers, but quickly became bored with the 
lessons. A few teachers, though, found that their students 
enjoyed the lab, and thought the software was fun. 

Part of the reason for students' boredom may be attributable 
to their having been placed incorrectly and having inappropriate 
lessons chosen for them. Both of these tasks are time-consuming 
and require detailed knowledge about students and the I.L.S. 
software. The lab manager explained that many teachers were 
frustrated because they wanted time to sit with her and discuss 
their students. They also wanted time to explain what they were 
doing in their classroom curriculum so that the lab manager could 
select appropriate software lessons for the students. The only 
time they had to meet with the lab manager, however, was when 
they were with the students in their lab periods. 

Another ongoing issue in School B related to the time it 
takes to learn how to use an I.L.S. effectively. As explained 
above, teachers were generally dissatisfied with their initial 
training for use of the lab. Several of these teachers noted 
that after students had started using the lab, however, they 
realized that they had learned enough for the time being, and 
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that learning more at first would have been an overload. It was 
only after half a year of student use of the lab that' some 
teachers felt ready to go on to learning how to do more than have 
students use the individualized lessons. They were now ready to 
discuss in greater depth how to integrate the use of these 
computers into their classroom curriculum, and to learn how to 
use applications like a writing program that is on the system. 
These teachers indicated that the staff developer from Waterford 
was interested in working on these aspects with them, and was 
planning sessions for this purpose. 
Academic Achievement 

Standardized achievement tests in reading and mathematics 
were administered to all students in spring 1994. As indicated 
in Table 1, there were no significant differences in normal curve 
equivalent scores in reading between program participants and the 
rest-of -school population at any grade level. 

Similarly, in math, and as shown in Table 2, although there 
was a trend favoring the rest-of-school population for the third 
grade (£=2.1; d£ 1,2060? b<0.15), there were no significant 
differences between Titan students and the rest of the school in 
grades 4 or 5. 
CAPITAL SCHOOLS 

Titan was responsible for four elementary schools in the 
capital funds phase of the I.S.L. project. Because the labs were 
ready for operation so late in the year, only setup and instal- 
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Table 1 



Comparison of Spring 1994 Reading Normal Curve Equivalent 
Scores of Titan Program Students and the Rest of School 







Program Students 
N Mean S.D. 




Rest 
N 


of School 
Mean S.D. 


P 


Grade 


3 


95 


36.3 


20.1 


1, 


983 


44.6 


22.7 


NS 


Grade 


4 


127 


38.7 


21.8 


1, 


949 


44.7 


24 . 5 


NS 


Grade 


5 


103 


45.8 


19.8 


2, 


095 


47 .0 


22.2 


NS 



According to analysis of variance tests, there were no 
significant differences between the reading scores of 
program students and the rest-of-school population at 
different grade levels. 
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Table 2 



Comparison of Spring 1994 Mathematics Normal Curve Equivalent 
Scores of Titan Program Students and the Rest of School 



Program Students Rest of School 







N 


Mean 


S.D. 


N 


Mean 


S.D. 


P 


Grade 


3 


92 


43.3 


19.5 


1,985 


44.6 


24.0 


0.15 


Grade 


4 


124 


43.6 


19.7 


1,951 


49. 7 


22.8 


NS 


Grade 


5 


101 


50. 2 


21.5 


2,089 


53. 1 


22.1 


NS 



According to an analysis of variance test, there was a 
slight trend toward rest-of-school scores being higher 
than program students in grade 3 (£=2.1; df 1,2060; 
E<0.15, although there were no significant differences 
in the other grades. 



lation, initial training issues, and other initial decisions are 
described here. 

Setup and Install ation of I.LS. Labs 

Administrators in the capital phase schools reported that 

district office personnel (usually the district's computer 

coordinator) had made decisions about which integrator to use. 

As far as administrators knew, their districts had chosen C.N.S. 

either because they had already worked with them or because 

C.N.S. had the most open management system. In one case C.N.S. 

was chosen because of the company's policy of not charging extra 

at a later date for adding more computers to the network ♦ 

The labs in the four schools were not ready to use until 

March or early April 1994. Both administrators and lab managers 
reported problems with delivery and installation of the lab 
equipment. In several cases, Apple took a long time to ship the 
computer equipment because it was waiting for a new line of 
Macintoshs (the LC575) to be ready. This new version was an 
upgrade that Apple provided at the same price as the older 
computers would have cost, but it meant a considerable delay in 
getting started. One school reported that it took a long time to 
get the security devices delivered and installed* According to 
the lab managers, none of these problems were caused by C.N.S. 
To the contrary, lab managers and administrators praised C.N.S, 
for its efforts to facilitate and coordinate delivery and 

installation of equipment. 
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Initial Training and Ongoing Support 

At the four schools half of the administrators and all of 
the lab managers interviewed had participated in some of the 
initial training. Topics covered included installing and using 
the system software, enrolling and placing students in the 
system, using the lesson software, troubleshooting hardware 
problems, and generating and analyzing reports. The amount of 
time spent in this training varied widely, totalling three hours 
for one of the schools, . >out six hours for another, 18 hours for 
the third, and 36 for the last. 

Lab managers in three of the schools reported being highly 
satisfied with the amount of training they received, the topics 
covered, the manner in which it was presented, and the materials 
they received. The other school, in which the lab manager was 
not as satisfied, was the one which had received the least 
training. Further, the lab manager in the school with the second 
least amount of training time was unhappy about aspects of the 
initial training. Reasons for this dissatisfaction will be 
examined below. 

The two schools that received at least several days of 
initial training seemed happy with it in all respects. One 
manager said that she "learned so much" from both the C.N. 3. 
representative and the Water ford staff developer, and indicated 
that the C.N.S. representative was "very responsive and very 
professional." The other was very pleased with the Waterford 
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staff developer, who conducted between three and four days of 
workshops with all the teachers in the school. 

The lab managers at. the other two schools reported problems. 
One said that she knew a lot about computers already, so training 
was not so important for her, but that teachers in the school 
weren't given enough initial training. Before the students 
started using the lab, the staff had half a day with the 
Water ford staff developer. Unfortunately, the system went down 
at the beginning of this session, and by the time it went back up 
there was only an hour left, and the staff developer spent most 
of this time talking about Waterford, barely getting to using the 
computers. Because of this, the teachers were very angry. Since 
then, when the staff developer comes to the school, according to 
the lab manager, she circulates and talks to the children. She 
is, however; planning to have two sessions with teachers in the 
future . 

In the school that received six hours of initial training, 
the situation was different. Here, the C.N.S. representative 
worked with the manager on one day, and students started coming 
the next. Unfortunately, when the first class arrived a system 
error occurred, with nobody on hand to fix it, and the lab could 
not be used that day at all. Since this time the staff developer 
from Waterford doesn't seem to have anything in mind to cover 
when she visits the school and just asks what the lab manager 
wants to know. 



General Issues Identified b y the Schools 

These two schools identified several issues involved in the 
initial I.L.S. training • One involves the amount of staff 
development time teachers and lab managers need before children 
start using the lab. Although lab managers are often already 
comfortable with computers, this is not usually the case with 
other teachers, who feel the need for considerable guidance and 
instruction before they are comfortable* In this regard, it is 
significant that the two schools that received the least amount 
of initial training time both reported dissatisfaction with the 
training • The material covered in the shorter period was not 
enough to ensure the smooth operation of the lab without the 
presence of an outside expert. 

Another issue relates to who should provide the initial 
training for lab managers and teachers. The integrator in this 
case split a variety of training and support roles between C.N.S. 
and Waterford. C.N.S. was responsible for all technical support, 
including installation and setup, and problems with the hardware 
and software, while Waterford was responsible for educational 
uses of the software, and connecting software lessons to other 
curricula. Both the technical and the educational expertise are 
necessary to make the project a success. In this instance all 
four capital schools praised the C.N.S. representative lavishly 
for his technical support, his responsiveness to their problems, 
his professionalism, and his manner of relating to them, while 
two of the schools complained about the lack of educational 
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support. Not enough time was dedicated to working with teachers 
before lab operation began, and in one case only C.N.S. provided 
this initial training. Lab managers and administrators indicated 
that this problem may have been caused by the late starting date, 
which interfered with staff developers' schedules. 

A third issue involves what teachers need to know before 
they start using the lab. As with School B in the previous 
section, teachers can feel the need for an amount of initial 
training which they later realize they did not need at that time. 
They may later realize that they did not really need to know very 
much at first, or that a competent lab manager is what an I.L.S. 
lab really needs. The issue of perceived vs. actual need may 
thus cause unhappiness at first, but may be more or less resolved 
later, and when teachers have used the system for a while, they 
may then be ready to go more deeply into how to use the system. 
Perceptions of the I.L.S. System So Far 

School personnel also had opinions about the operation and 
effectiveness of the I.L.S system which had been in operation in 
their schools for nearly two months. 

Two schools reported that the system had gone down once or 
twice, but that C.N.S. had come very quickly to restore it to 
operation, and two of the schools indicated that they had begun 
generating class reports from the system, as well as some 
individualized student logs. Lab managers were not sure if 
teachers were using these reports yet. 
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Lab managers at three of the schools were happy with the 
I.L.S. as a whole. Although students and teachers were still at 
the early stages of using the software, all managers thought that 
using such software would be academically beneficial to students* 
One explained that "The kids love it. I like the spftware. The 
different games drill on necessary skills." Another said that 
"Teachers think the lessons are good follow-up for their 
classroom work. I think it's a terrific program and it's going 
to do wonders for the school." 

Two of these schools were also using the lab for more than 
the C.N.S. lessons. One school had a computer club which had 
written and published a newspaper for the school. In the other, 
teachers were beginning to use the lab for their own writing. 

The fourth school was not as happy with the I.L.S. software. 
* This school already had an extensive background using computers 
as tools for creating publications, creating databases, using 
spreadsheets, and telecommunications. Furthermore, admini- 
strators had supplied each of the upper grade classrooms with 
three or four Macintoshes, and the labs in the lower grades were 
in the process of networking with the other computers. Teachers 
were already familiar with much commercial software, and the 
school in fact already had some of the software that C.N.S. used. 
According to the lab manager, the school thought it was getting 
diagnostic/prescriptive software with the I.L.S. , and were 
disappointed with its diagnoses. Because of the school's history 
with computers and orientation toward using them, teachers at 
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this school did not want to use computers as what the lab manager 
called "electronic workbooks." The lab manager explained that, 
"The whole technology program in this school was implemented with 
respect for individual teachers, and each teacher is doing things 
in a different way." This school was unaware of what kind of 
software they were choosing when the district selected Titan as 
the integrator. However, the lab manager emphasized that they 
appreciated the efforts of C.N. S. in performing all its roles, 
and that it was the I.L.S. itself which they did not like. 
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III. CONCLUSIONS AND RECOMMENDATIONS 

Two kinds of findings are reflected in this report. One 
concerns the performance of Titan as an educational systems 
integrator, while the other focuses on the hardware and software 
that make up the I^L.S. itself. 

In this project, Titan delegated its role as educational 
systems integrator to C^N.S., which assumed responsibility for 
installing and maintaining hardware and software, and to a large 
extent training lab managers how to use it. One of the clearest 
findings of this study was the esteem in which C.N.S. itself was 
held by every school. They were unanimous in their opinion that 
the C.N.S. representative was professional, knowledgeable, 
responsive to problems and needs, and a pleasure to work with. 
There was only one area in which some complaint about C.N.S.' 
performance was voiced, and that was the length of time it 
occasionally took C.N.S. to get to one of the schools to fix a 
system problem. This is a serious concern, since time spent 
waiting for service can mean time during which students cannot 
use the computers. It should be noted, however, that the demands 
on a limited number of individuals by several schools can often 
be taxing, and that this complaint was voiced in only one of the 
six schools. 

C.N.S* had subcontracted the teacher training aspect of the 
program to Waterford, which received mixed reviews. Half of the 
schools were very satisfied with the training they received, 
while the other half voiced dissatisfaction about initial 
training in several areas: the amount of time allocated for 
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training; the topics covered; and, in one case, the trainer's 
manner of presenting the topics. Also, these same schools were 
not happy with the trainer's ongoing work with school staff. 

Respondents 1 comments shed light on several aspects of the 
training process, and raise several questions about what is 
necessary for successful implementation of an I.L.S. project. 
First, what kinds of training are necessary for teachers and lab 
managers to use an I.L.S. , and who is responsible for providing 
it? On the one hand, lab managers and, to some extent, teachers, 
need technical training in how to operate the computer, use the 
system software and the students' software, and troubleshoot 
hardware and software problems. On the other hand, they need to 
understand how to use the software for educational purposes. The 
data seem to indicate some confusion about who was responsible 
for which kind of training. C.N.S. was clearly responsible for 
the main technical aspects of it, while Waterford dealt with most 
educational matters. Waterford 's role in most schools was also 
to show staff how to use the computer and the system software. 
In one school, however, a Waterford staff developer conducted all 
the training before students started using the lab, and in 
another the Wauerford trainer did hardly any training. Also, 
respondent* in two schools complained that, after students 
started arriving, the Waterford trainers didn't seem to know what 
to do on their weekly visits. 

A second aspect involves the timing of the training. 
Several of the schools got a later than expected start in 
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implementing the project , and in these schools respondents tended 
to complain about the amount of training they received. It is 
possible that, in these cases , had it been feasible to follow the 
original schedule, more initial training would have been 
provided. Also, in some cases technical problems with machinery 
interfered with part of the original training time. 

Timing was also important in the question of what should be 
taught at which stage of the project. What do teachers need to 
know to start using an I.L.S. with students, and what could they 
learn later? Since a lab manager was on hand for technical 
support in almost every case in this project, it is doubtful that 
the teachers needed to know much about operating the system. In 
fact, several teachers remarked that, although at the time they 
wanted more initial training than they received, upon reflection 
they realized that they did know enough to begin using the I.L.S. 
Later on, the kind of training they wanted tended to involve 
using various computer tools such as word processors, and not 
necessarily the I.L.S. drill software. One implication of this 
funding is that perhaps teachers' original assessments of their 
training needs may not be consistent with what they really need 
to learn. Another is that the trainers should be clearer about 
what teachers need to know ac the beginning, and what can be 
covered later. 

A third area involves the role of the lab manager. One 
conclusion that emerges from data gathered in this study is that 
the lab manager was a key person in insuring the success of the 
project. The lab manager was typically someone who already 
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possessed some knowledge, and often considerable knowledge, about 
computers. Trainers from C.N.S. or Waterford spent a good deal 
of time with managers to ensure that they could operate the 
system software, enroll students, change placements, and so on. 
As the project was implemented, teachers relied on these lab 
managers, since they were the ones trained to use the system. 
Several schools, in fact, voiced doubts about whether the project 
could have succeeded as well as it did without a knowledgeable 
lab manager or, in one case, a subject-area teacher very 
experienced with computers. 

The role of the lab manager is relevant to the issue of 
teachers' preparation and ongoing training for using the lab. 
Teachers were more or less satisfied with their initial training, 
but as students started using the lab, smooth implementation 
seemed to depend on how well lab managers were prepared. At all 
stages of the project, but especially at the beginning, the 
crucial factor was knowing how to use the I.L.S. and keeping it 
working correctly. Teachers could of course help students with 
the lesson software, and sometimes problems occurred which lab 
managers could not handle, but the bulk of the activity and 
responsibility involved lab managers making sure the I.L.S. was 
delivering software lessons to students. 

Further, teachers in several schools reported that when they 
began thinking of how to integrate I.L.S. lessons with their own 
classroom curriculum, they relied on the lab manager for help 
rather than a teacher trainer. In some schools, the Waterford 
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staff developer worked with teachers in thinking through this 
integration process, but not in all of theau Two school' 
reported that, after the initial training, the staff developers 1 
visits to the school involved circulating and helping students. 
In the two grantback schools, which operated through most of the 
year, teachers worked well with the trainer in one, but in. the 
other they seemed to rely solely on the lab manager. 

Taken together, these findings present a somewhat 
inconsistent picture of the training provided to teachers. The 
amount of initial training time varied widely among schools, in 
some schools teachers were satisfied with the topics covered, 
while in others they were not. In some cases, teachers could be 
satisfied with little initial training because they in fact did 
not need to know very much. Teachers 1 level of satisfaction 
sometimes seemed to depend on how much they thought they needed 
to know, rather than what they actually needed to know. Finally, 
in some schools the staff developer worked well with teachers 
after the initial training, while in others there was not much 
interaction. 

Regarding perceptions of the I.L.S. itself, findings are 
also mixed* Th* majority of schools seemed happy with the 
system, but some expressed reservations. 

One area of dissatisfaction was schools 1 involvement in 
decision making about the I.L.S. project. Most schools reported 
they had little or no input into initial decisions about what 
integrator of software to use, and teachers had even less say. 
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Some teachers and administrators expressed skepticism about the 
project because the school was not involved in planning it, 
because they saw it as imposed on them, or because they doubted 
that outsiders would know what their schools needed. The data 
show that these fears were overcome after integrators started 
working with schools. Many comments, however, reflect the lack 
of fit between I.L.S. software and school curriculum, and some 
respondents believed that this could have been alleviated if 
they, or the school as a whole, had been involved in making 
decisions about software. 

Opinions about the software itself were also mixed. 
Although many teachers thought the lesson software was beneficial 
to students, others said students found it boring or 
uninteresting. Also, as noted above, a considerable number of 
teachers said that the software did not fit their own schools 1 
curriculum, so that they had difficulty integrating it into 
classroom activities. Finally, one of the capital schools found 
that the type of drill software on the I.L.S. system did not 
match its educational goals. 

Student achievement scores showed no significant differences 
in reading between program participants and the rest-of-school 
population. Mathematics scores indicated a trend towards 
difference* in grade 3. However, this trend favored the rest-of- 
school population, not program participants. 

Based on these evaluation findings, OER makes the following 
recommendations : 
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Reexamine the teacher training process to clarify such 
aspects as how much and what kinds of training teachers need 
before students begin to use the I.L.S. system, and what 
kind of ongoing support teachers need to ensure successful 
student use of the lab* 

Clarify more completely the roles of CN.S. and Waterford in 
delivering technical and educational support to schools. 

consider how the program expects schools to integrate use of 
the I.L.S. lab with the rest of their school curricula, and 
systems integrators will work with schools to effect this 
integration. 



